The influence of task activity and the learner’s personal 
characteristics on self-confidence during an online 
explanation activity with a conversational agent 


Yugo Hayashi 
Ritsumeikan University 
2-150 Iwakura-cho,|baraki 
Osaka,567-8570,Japan 
y-hayashi@acm.org 


ABSTRACT 


This study investigated the factors underlying the estima- 
tion of learner self-confidence during explanations with a 
conversational agent in an online explanation task. Based 
on reviews of previous studies, we focused on how factors 
such as the learner’s task activities and personal character- 
istics can be predictors. To examine these points, we used 
an online explanation task, which was run by a conversa- 
tional agent for students in a classroom on information pro- 
cessing psychology (n=99). We asked the participants to 
make text-based explanations to the agent in a question- 
and-answer (Q&A) style, and clarified a particular concept 
that was taught in a previous lecture in the class. The re- 
sults show that an increase in the amount of actual task work 
for explanations and personal characteristics (such as social 
skills) helped to predict higher self-confidence. The findings 
have implications not only for knowledge of how such factors 
might influence learner self-confidence in an online explana- 
tion task, but also for the design of online tutoring systems 
that can automatically detect learner confidence using these 
variables, and facilitate learning adequately based on such 
data. 


1. INTRODUCTION 


Networked learning such as the use of massive open online 
courses (MOOCS) and tutoring systems, which include so- 
cial networking services (SNS) has seen many advances in 
recent years and has become a popular way of supporting 
learning through social interaction. Such environments al- 
low learners to interact with each other through conversa- 


tion, and have drawn the attention of many socio-constructionists 


in the field of learning science. Numerous investigations in 
this field focus on discussion boards [5, 25], and an emerg- 
ing number of studies have examined the technological side 
of research. Moreover, these studies have explored how to 
detect the learner’s conversational behavior. Researchers in 
artificial intelligence education (AIED) have been investi- 
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gating the use of conversational agents (CAs) in online envi- 
ronments [20] and have explored the use of agents that play 
the role of peer learner, whereby they interact socially as dis- 
cussants in a serious game-based environment [20]. Some re- 
search on online tutoring systems examines the use of agents 
that play the role of the student, whereby learners absorb in- 
formation through teaching the agents [16, 17]. One of the 
most important points of learning by teaching is that the 
learner can reflect on his ideas by observing his externalized 
thoughts. In the context of social learning, metacognitive 
abilities might help him identify the perspectives of other 
learners/agents to establish shared knowledge and success- 
fully coordinate with one another. 


Despite concerns surrounding the effects of social learning 
on social coordination skills and metacognitive abilities, not 
many experimental investigations have explored the learner’s 
task efficiency and metacognitive process, such as confidence 
during interactions with a CA. Our study centers on the 
learner’s metacognitive capacity; for example, in relation to 
confidence evaluations in an explanation activity with a CA. 
We investigated how the learner’s task activity and personal 
characteristics impact his confidence level during tasks, and 
propose a model to understand learner confidence during on- 
line tutoring with an agent. We also discuss how our model 
could predict learner confidence, and subsequently develop 
an automated tutoring system that can collaboratively re- 
spond based on learner confidence. 


1.1. Conversational Agents and their use in on- 


line Learning 
The number of studies on computer-based learning that em- 
ploys intelligent tutoring systems has grown rapidly over the 
past three decades/14, 27]. Advances in language technology 
have enabled the development and use of CAs, which can 
act as peers learners or mentors, and have made progress in 
terms of facilitating learning activities[10, 8]. Initial studies 
focused on the use of embodied CAs that act as educational 
companions or tutors and facilitate the learning process as 
it relates to motivation[4]. Moreover, recent research has ex- 
amined the implications of such technology on learning gains 
through learning by doing [1]. Many studies investigate the 
use of agents capable of handling natural conversation; these 
agents are developed based on conversational dialog models, 
and have demonstrated the successful use of tutoring in so- 
cial interactions. One example is AutoTutor, a system that 
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allows students to engage in conversations for their projects. 
Recently, more advanced online tutoring systems, such as 
Operation ARIES!, have employed CAs [20] where learners 
absorb information through web-based tasks in which they 
talk with CAs. Other tutoring systems have begun to in- 
corporate elements such as SNS [13]. In such cases, learners 
can interact with other learners and CAs; they must use 
metacognition to monitor their own perspectives as well as 
those of their peers in order to better coordinate with one 
another. Many important psychological issues have not yet 
been explored in depth; for example, how learners develop 
their confidence by reflecting on activities in such an envi- 
ronment. In the next section, we will look at some of these 
points based on reviews of related studies. 


1.2 Self-confidence and learning 

The 2015 report of the Programme for International Stu- 
dent Assessment (PISA) an initiative of the Organisation 
for Economic Co-operation and Development (OECD) iden- 
tifies several types of skills such as prior knowledge, personal 
characteristics, collaborative capacity, and problem-solving 
skills. These abilities were assessed using pedagogical CAs, 
which acted as peer learners and tutors. The report men- 
tions self-monitoring as an important skill because learners 
must be able to keep track of how their abilities, knowledge 
and perspectives affect their interactions with other agents 
in relation to the task at hand [23]. Monitoring skills can 
be detected through evaluations of self-confidence; this issue 
has been broadly examined in cognitive psychology. 


Cognitive psychology research has a long history of study- 
ing metacognition, such as self-monitoring of task efficiency, 
which is deeply linked to performance [19]. According to the 
literature, problem-solving involves conscious, step-by-step 
observation of one’s problem-solving behavior. Throughout 
this process, one can estimate the likelihood of the ongoing 
task having success or failure. In terms of learning activities, 
high confidence is known to reflect higher quality mental rep- 
resentations of a task, and is associated with long-term recall 
[7]. In this sense, a hypothesis can be deduced, such as that 
the actual task activity might facilitate the learner’s mon- 
itoring; for example, regarding the self-evaluation of one’s 
confidence about a task. Interestingly, some educational 
psychology studies have revealed that self-assessments of 
learning achievement are negatively correlated with learning 
performance [9]. One explanation for this outcome might be 
that learners have inherent cognitive limits that hinder si- 
multaneous monitoring and execution of a task. They might 
also have individual differences in terms of their capacity to 
self-monitor. Additional types of individual skills that can 
be captured by self-assessments might play a role in self- 
monitoring. In this context, we investigated participants’ 
ability to self-monitor their confidence about a task activity. 
Next, we analyzed the relationship between self-monitoring 
and the personal characteristics, which might also affect con- 
fidence level. 


1.3. Personal characteristics and Learning 

Along with concerns raised in the previous section about per- 
sonal characteristics, recent reports have shown that quali- 
ties such as attitude, interpersonal skills, personal traits, and 
motivation can influence individual learning activities[23]. 
Studies examining such personal features have shown that 


these factors indeed influence learning; for example, when it 
comes to thinking style [26]. In the context of this study, 
where learners interact socially with an agent, it is impor- 
tant to focus on the learner’s personal qualities as they re- 
late to social interaction and communication skills. Some 
research has explored the use of Big-Five questionnaires 
[15], which center on personal characteristics, such as so- 
cial skills. Previous studies have indicated that learners 
with poor social skills might have lower collaborative per- 
formance [21]. Other studies by [12] have investigated how 
learners’ skills influence their performance during an online, 
concept-learning tutoring task with a pedagogical Conver- 
sational Agent (PCA). During in this task by [12], learners 
were guided by a PCA that helped them formulate their 
explanations of a key concept taught in a large-scale class. 
The results show that learners with higher social skills per- 
formed better on explanation activities with the PCA. Tak- 
ing this into consideration, personal characteristics such as 
social skills will also influence metacognitive states, which 
are related to task performance. Thus far, no investigations 
have delved into the relationship between social skills and 
self-confidence; however, this study does. Based on this, we 
focus on a particular situation whereby most studies using 
agents have not yet fully examined the influence of personal 
characteristics on learning activities. 


1.4 Goal and Hypothesis 


This study investigates how the learner’s task work influ- 
ences metacognition of his/her work, and consequently, self- 
confidence. Furthermore, we examined how personal charac- 
teristics, which are considered important for inter-personal 
interactions, impact both the task activity and the learner’s 
metacognition of the task. To explore these points, we used 
an online explanation task where we asked learners to give 
explanations to a social CA in a Q&A style, and to chat 
about a particular concept that was taught in a previous lec- 
ture. Based on reviews of previous research on learning ac- 
tivities and metacognition, we hypothesized that an increase 
in the amount of actual task work, such as giving many ex- 
planations to an agent, would enhance self-confidence about 
one’s work (H1). For our second goal, we focused on the 
relationships between personal characteristics and work on 
explanation activities, as well as the learner’s metacognition 
of that work. We posited that higher interpersonal skills 
would increase the number of actual explanation activities 
in relation to the social agent (H2-a), and would also enable 
metacognition of the student’s explanations (H2-b). In the 
next section, we will demonstrate how we analyzed these 
points. 


2. METHOD 


2.1 Participants and conditions 

Ninety-nine (Mage: 20.52, SD: 1.60) Japanese university 
students majoring in psychology participated in this study. 
The students, whom we call learners, were taking a lec- 
ture class on information processing psychology in 2014 and 
used the system as part of their coursework. The learners 
were taught about 30 basic concepts of human information 
processing such as top-down processing, neural networks, 
Bayesian models, and expert systems. 


2.2 Procedure 
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After the participants attended lectures about the basic con- 
cepts taught in class, they took part in an online tutoring 
task that was valid for two weeks. They logged into the 
web system using their ID and password, and worked on 
the task based on their personalized page. They could only 
access the system on campus using the computer terminals 
located there. Only members of the class were registered 
in the system and were assigned to groups consisting of 4- 
5 students. In each group, the participants worked on the 
same materials, and the system provided them with updated 
information about their fellow members. 


The aim of the task was to facilitate learner’s self-explanations|6] 


of the basic ideas they learned in class by conversing with 
social agents though online texts. As they began the task, 
the agent appeared on their screens and asked them ques- 
tions about a specific concept. The questions consisted of 17 
types such as: “Can you explain the key term regarding how 
it functions?” “How do you use it in your daily life?” and 
“Can you think of a concept similar to this one?” Learners 
were able to restart and continue the task, even if they ter- 
minated it during the Q&A session. After they answered one 
question, the page switched to an assessment page, and the 
system asked them to assess their confidence level. As will be 
explained in the following section, this was done to measure 
the degree of self-confidence. Afterward, learners received 
feedback from the social agent, along with examples from 
other members when inputs were entered into the database. 
If there was no updated information from classmates, the 
system used a database from the previous year instead. As 
learners finished answering all 17 questions, they completed 
the task. This activity lasted an average of 30 minutes. 


2.3. System 


The system was operated on an Apache web server via a 
CentOS server. The scripts of the web pages were written 
in PHP, JavaScript, HTML and CSS. MySQL was used for 
the database. This system is a modified version of [11] and 
is called “ Web-based Explanation Support with Conversa- 
tional Agent” (WESCA). The system has a database that 
manages thirty different key terms that were selected from 
the class material; one was assigned to each of the learners 
according to their ID numbers. The agent in the system 
responds to the learner’s text sentences and the number of 
questions based on the bag-of-word method. The system 
can also retrieve other members’ answers (using them as ex- 
amples) from the logs based on year, and data from previous 
years if there is no updated information. The system also 
features social awareness functions such as evaluating the 
other learners pushing the “like” button. The current ver- 
sion does not have any functions to show learners how many 
likes they have received during the task. 


2.4 Measures 


This study focuses on three factors: (1) degree of self-confidence 


while interacting with the agent, (2) the amount of interac- 
tion with the agent, and (3) the effects of personal charac- 
teristics on social skills. In the following section, we describe 
the types of measures that we used to capture these factors. 


2.4.1 Meta cognition: Self-confidence 
To capture learner self-confidence during the participants’ 
explanations, we collected assessments based on confidence 


level about the explanations for each Q&A session with the 
agent. Learners were required to input their self-confidence 
level based on a seven-point scale (-3: not very confident 
to 3: very confident). As with the number of interactions, 
we analyzed the average level of confidence for each individ- 
ual, and used these levels as representative values for each 
participant. 


2.4.2 Number of interactions: The amount of words 


used to explain 
We calculated the number of interactions based on the num- 
ber of words that the learners input while responding to 
the agent. For each individual learner, we used the aver- 
age number of words that were input into the system as a 
representative value for the number of interactions with the 
agent. 


2.4.3 Personal characteristics: The autism spectrum 


quotient (AQ) score 

We assessed the degree of social communication skills based 
on the questionnaire, which was originally developed in [3] 
and translated into Japanese. This questionnaire appraises 
social skills based on the autism spectrum quotient (AQ) 
and was originally was used to investigate whether healthy 
adults had symptoms of autism. The questionnaire consists 
of 50 questions covering five different domains associated 
with the autism spectrum: (1) social skills, (2) attention 
switching/tolerance for change, (3) attention to detail, (4) 
communication skills, and (5) imagination. For each ques- 
tion, learners assessed how strongly they felt about them- 
selves on a five-point scale (1: Doesn’t match, 5: Does 
match). For example, a question about social skills would 
be, “I like to do activities that require interacting with oth- 
ers.’” The higher the score, the lower the learner’s degree of 
autism, which indicates strong social communication skills. 
For each learner, we calculated the five factor scores of do- 
mains (1) to (5) using factor analysis, and used this as the 
representative value for analysis. 


3. RESULTS 


To examine our two hypotheses, we first explored how learn- 
ers’ explanations that they gave to the agent influenced their 
self-efficiency. For this point, we investigated the relation- 
ships between (1) the number of explanations given to the 
agent and the degree of learner self-confidence regarding the 
achievement of the activity. Then, we looked at how individ- 
ual characteristics (such as social communication skills) in- 
fluenced both the number of explanation activities and self- 
confidence. For this aspect, we analyzed (2) the relationship 
between the AQ scores and the number of explanations and 
confidence levels. 


3.1. Explanation activities and self-confidence 
First, we conducted a correlation analysis using the Pearson 
correlation coefficient to identify any relationships between 
the two variables, as well as the average number of words 
used during the explanation activity, and the average confi- 
dence level about the explanation given to the agent. The 
findings show that there were significant relationships be- 
tween the two variables (r = 0.211. p < .05). Figure 1 
describes the correlations between the two variables. Next, 
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Figure 1: Relationship between learner’s confidence 
and typed words. 


Table 1: Results of correlations between personal 
characteristics and number of words 


# of words 
social skills 0.051 
. attention switching -0.023 


. attention to detail 0.149 
. communication skills 0.076 
. creativity 0.034 


oe wwe 


we explored how explanation activities influenced confidence 
level by conducting a single regression analysis. We em- 
ployed the evaluations of confidence as the dependent vari- 
able, and number of words used during explanations as the 
independent variable. We used the forced entry method to 
perform the analysis and acquired the regression equation 
with the coefficient of determination (R?=0.035 by p < .05). 
These results suggest that the actual performance of inter- 
actions (such as explanation activities) facilitates metacog- 
nition, thus supporting hypothesis H1. 


3.2 Personal Characteristics 
3.2.1 Personal Characteristics and explanation ac- 
tivities 

Next, we analyzed the correlations between the scores of the 
five domains and the types of words to see how the personal 
characteristics considered by the AQ questionnaires related 
to task activity. More specifically, we examined the correla- 
tion between the number of words used for the explanations 
and each of the five AQ domain factors: (1) social skills, (2) 
attention switching/tolerance for change, (3) attention to 
detail, (4) communication skills, and (5) imagination. Table 
1 depicts the correlations between the variables. 


The outcomes of the analysis of the Pearson correlation co- 
efficient revealed no significant links between any of the AQ 
categories. This indicates that personal qualities captured 
from the AQ scores do not have any influence on explana- 
tion activities with the agent. This shows that hypothesis 
H2-a was not supported. 


Table 2: Results of correlations between individual 
characteristics and learner’s confidence 


leanrer’s confidence 


1. social skills 0.312 
2. attention switching 0.211 
3. attention to detail -0.164 
4. communication skills 0.170 
5. creativity -0.025 
e communication 
@ attention 
5.00 e + e @ social skill 
e creativity 
© switch attention 
4.00 + ° 
e 
g $ 
‘oe me ; 
8 3.00 : is <_ _ 
. 
. 
2.00 
e 
e e 


1.00 


Confidence 


Figure 2: Relationship between each AQ score (Y- 
axis) and learner’s confidence( X-axis). 


3.2.2 Personal Characteristics and self-confidence 

Next, to see how personal characteristics considered by the 
AQ scores were related to confidence, we looked at the corre- 
lations between the AQ scores and confidence level. For each 
of the five AQ factor scores, we explored the correlation with 
confidence level. Table 2 shows the correlations between the 
variables. The findings of the analysis of the Pearson correla- 
tion coefficient revealed significant links between confidence 
and three variables from AQ: (1) social skills (r = 0.312. 
p < .01), (2) attention switching/tolerance for change (r = 
0.211. p < .05), and (4) communication skills (r = 0.170. p 
< .05). To see if personal characteristics considered by the 
AQ scores influenced the number of explanation activities, 
we conducted a multiple regression analysis. Figure 2 shows 
the outcomes of the correlations between the two variables. 
We employed the number of words as the dependent variable 
and the five AQ factors as the independent variables. We 
used the forced entry method to perform the analysis, and 
acquired a regression equation with the coefficient of deter- 
mination R?=0.137 by p < .05. Table 3 shows the summary 
of the multiple regression analysis. These findings demon- 


Table 3: Summary of the mutiple regression analysis 
regression coefficient B 


1. social skills .368 
2. attention switching .044 
3. attention to detail -.041 
4. communication skills .004 
5. creativity -.203 
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Figure 3: Overall results of AQ score amount of ex- 
planations and self-confidence. Indices’s indicate the 
regression coefficient B. 


strate that only the variable of social skills influenced learner 
evaluations of confidence. This indicates that personal char- 
acteristics influence learners’ metacognition of their learning 
activities, thus supporting hypothesis H2-b. Figure 3 por- 
trays the summary of the results, including path variables. 
This figure shows the model of how personal characteristics 
and actual task work facilitate self-confidence. Our data 
analysis suggests that this model could potentially predict 
learner self-confidence, which we will discuss further in the 
next section. 


4. DISCUSSION 


4.1 Developing an automatic tutor to detect 


learner self-confidence 
Our results show that self-confidence was related to the task 
activity, as well as personal traits (such as social skills). 
Considering these findings, the actual amount of words in- 
put and previously self-evaluated personal scores can help 
predict confidence level. Therefore, we can use this model 
to develop systems that can become aware of the learner’s 
subjective states. We can also employ it to design peda- 
gogical agents that could prompt learners to request help 
or encourage those with low confidence. As discussed ear- 
lier, learner self-confidence is highly related to task perfor- 
mance|7]; identifying learners’ cognitive states might facil- 
itate self-efficiency [2] during the task, which could result 
in higher learning performance. Discussions about learn- 
ing performance could go beyond the topic of this paper. I 
would like to show how the proposed model could be used 
to automatically predict learner self-confidence during the 
task. For this investigation, we used machine learning to 
see how the categorical factors that were extracted from the 
previous analysis might be optimal for detection. We used 
linear discriminant analysis (LDA), using confidence as the 
dependent variable and the number of words and social skills 
score as independent variables. Confidence was labeled as 
a binary of high/low based on the median of the acquired 
data set. The results of the LDA show an accuracy rate of 


66.7%, which indicates a relatively high validation of cate- 
gorization. There have been recent attempts to detect and 
model self-efficiency in tutoring systems/18]. However, not 
many studies focus on personal characteristics as predictors. 
In this sense, our model could provide a new way to capture 
learners’ subjective states. However, as noted above, more 
integrated investigations should be carried out, along with 
an analysis of learners’ performance during the explanation 
activities. To do so, we should evaluate learners’ output 
messages and see how they relate to the variables acquired 
in this study. 


4.2 Motivating learners via socialized feedback 


from the conversational agent 

The system used in this study features functions such as 
providing feedback about other group members’ explana- 
tions. Moreover, learners were able to assess each other’s 
explanations by clicking on the “like” buttons, as in SNS. 
These social functions are adequate for motivating learners 
and reducing the dropout rate. One of the methods used to 
facilitate learner self-efficiency in such educational environ- 
ments could be designed by providing feedback, such as how 
many “likes” they receive during their activities. Telling 
learners that they have been nominated as good explainers 
in the group is another way to motivate them. The CA 
can provide such feedback, as it is well-known that people 
can praise each other in human-computer interactions [22]. 
Related studies from our research group have been develop- 
ing systems through which students can request help online, 
as well as systems that support teachers in programming 
classes[24]. Learners in the classroom use the system and 
report the ongoing progress of their programming tasks. As 
they complete each task, an agent installed in the system 
contacts the learner and sends a request for him to help 
other classmates who are still stuck working on a problem. 
The system aims to increase learners’ self-esteem by approv- 
ing/selecting him to help his classmates. The study focuses 
on motivation when a learner becomes a teacher, as well as 
on learning in the domain of programming skills. In future, 
the system to be introduced in this current study might uti- 
lize such features, the goal being to encourage learners to use 
these types of help-requesting functions provided by CAs. 


5. CONCLUSIONS 


This study focused on self-confidence during explanations 
with a CA in an online explanation task. The study aimed 
to understand how the actual activity conducted during the 
task influenced the learner’s metacognitive state. Moreover, 
based on the literature on personality and individual differ- 
ences, we investigated how interpersonal traits related to so- 
cial communication could become predictors for the learner’s 
task activity and his metacognition of it. Using an online tu- 
toring system developed by [11], we collected learners’ activ- 
ity logs of explanations, evaluations of their confidence, and 
AQ scores. The results of the regression analysis revealed 
that increasing the amount of actual task work, such as giv- 
ing many explanations to a social CA, enhances learners’ 
self-confidence about their work, thus supporting hypothe- 
sis H1. The analysis of personal characteristics showed that 
social skills influence self-confidence (thus supporting H2-b); 
however, they do not influence the actual task work (H2-a is 
thus not supported). These outcomes indicate that personal 
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traits affect self-confidence in interactions with a social CA. 
These findings have new implications for designing tutoring 
systems that can assess and detect learner confidence during 
online learning activities. An additional analysis using ma- 
chine learning has also been conducted to investigate if the 
model suggested in this study could be used to automatically 
detect learner confidence and thus showed the effectiveness. 
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